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with  magnetic  anomalies  noted  by  various  symbols . 3 
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quadrant  is  shown  on  an  expanded  scale  on  the  left.  This  is  an  Analytic 
Signal  image  that  shows  a  concentration  of  targets  typical  of  a  target  bull’s 
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1.0  Introduction 


This  project  had  its  genesis  in  the  FY-2007  SERDP  Proposal  Cycle  as  proposal  07  MM04-007. 
The  Phase  II  Defense  was  briefed  to  the  Scientific  Advisory  Board  on  1 8  October  2006  and  was 
approved  for  funding  on  15  November  following  revision  to  incorporate  recommendations  of  the 
Advisory  Board. 

Following  the  sale  of  AETC  to  SAIC  in  November  2006,  the  project  was  awarded  to  SAIC  by 
HECSA  as  Contract  Number  W912HQ-07-C-0023  on  23  April  2007.  Because  SAIC  and  VLS 
were  partners  in  this  project,  work  could  not  begin  until  a  subcontract  was  in  place  with  YLS. 
Because  of  many  miss-steps  along  the  way,  the  subcontract  with  VLS  was  not  put  in  place  until 
September  2007.  Therefore  the  program  startup  was  delayed  by  a  half  a  year. 

The  Project  Plan  calls  for  applying  the  techniques  developed  during  the  previous  projects, 
UX1322  and  UX1455  to  the  vehicular  and  airborne  Wide  Area  MTADS  surveys  of  western 
desert  ranges.  In  project  UX-1455  we  demonstrated  that  using  machine  learning  techniques 
inherent  to  the  Feature  Analyst  software  it  is  possible  to  autonomously  identify,  with  high 
confidence  and  accuracy,  nearly  all  of  the  UXO  in  a  survey  dataset.  Furthermore,  we  showed  the 
technology  could  significantly  reduce  the  number  of  false  positives  using  a  two-pass  workflow  in 
Feature  Analyst  with  the  Target  Picker  and  Target  Ranker  modules  operating  (sequentially) 
separately  from  each  other.  The  objective  of  the  Target  Picker  is  to  independently  recognize  all 
anomaly  signatures  in  a  dataset  that  might  be  True  Positives  (ordnance).  The  selected  anomalies 
(referred  to  in  out  project  as  Regions  of  Interest  or  ROIs)  are  passed  to  the  Target  Ranker  where 
they  are  analyzed  again  to  classify  the  anomalies  as  ordnance  or  clutter. 

The  UXO  Toolkit  developed  for  Feature  Analyst  uses  a  two-pass  target  recognition  approach.  In 
the  first  pass  the  Target  Ranker  algorithm  emphasizes  the  anomaly  shape  in  the  analysis,  but  also 
incorporates  object  size,  color,  shadow,  texture,  pattern,  spatial  association,  and  signal  intensity 
as  feature  attributes  in  the  analysis  process.  During  the  second  pass,  the  Target  Ranker  algorithm 
evaluates  each  ROI  and  provides  a  probability  that  the  ROI  is  indeed  a  True  Positive  (Ordnance) 
target.  During  the  previous  project,  this  approach  was  used  to  analyze  the  Airborne  Survey  Data 
from  the  1700-acre  Badlands  Bombing  Range.  Following  training,  the  automated  analysis 
requires  -4  hours  (using  a  desktop  computer)  to  process  -10,000  anomalies  from  the  Range. 
The  human  analyst  required  -75  man  hours  to  accomplish  the  same  task.  The  automated  target 
picker  performance  was  equivalent  to  the  human-based  manual  selection,  and  the  Feature 
Analyst  classifier  correctly  specified  95%  of  the  UXO  with  80%  fewer  false  alarms  than  the 
human  analyst. 

Congress  authorized  a  Wide  Area  Site  Assessment  Pilot  Program  to  evaluate  UXO 
contamination  and  identify  areas  that  can  be  declared  ordnance-free  on  former  large  ranges.  The 
assessment  program  included  a  combination  of  LIDAR  and  Orthophotography  measurements 
from  high  altitude  platforms,  wide  area  survey  coverage  using  a  helicopter-based  magnetometer 
array  and  limited  transect  and  blanket  area  survey  coverage  using  a  vehicular-towed 
magnetometer  array.  The  surveys  of  interest  included  the  Pueblo  Precision  Bombing  Range, 
PBR-2,  the  Kirtland  Ranges  N1  and  N2,  and  the  Victorville  Range. 
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Project  MM-1570  has  two  objectives: 

Objective  #1:  Analyze  and  classify  all  magnetic  and  surface  anomalies  from  all  3  WAA 
Ranges.  Provide  a  UXO  probability  for  each  anomaly; 

Objective  #2:  Accomplish  Objective  #1  as  4  separate  tasks,  each  being  completed  and  reported 
before  beginning  the  next. 

Task  1:  Use  mapped  data  files  from  common  vehicle  and  airborne  survey  areas  and  ground 
truth  from  other  similar  sites,  to  analyze  and  rank  all  targets.  (External  Training) 

Task  2:  Use  the  same  data  and  selected  ground  truth  from  the  WAA  datasets  to  reanalyze  and 
rank  all  targets.  (Local  Training) 

Task  3:  Use  complete  ground  truth  from  dig  sites,  to  reanalyze  all  airborne  datasets  and  rank  all 
anomalies  (Complete  Ground  Truth,  with  data  censored  from  Bull’s  Eyes) 

Task  4:  Use  Feature  Analyst  (UXO  Analyst  and  LIDAR  Analyst  Utilities)  to  conduct  a  Joint 
Analysis  of  the  Orthophotography,  LIDAR  imagery,  and  Airborne  Magnetometry  Imagery 
datasets.  Provide  the  results  in  a  narrative  report. 

The  Program  Office  suggested  that  the  project  start  with  datasets  from  the  Pueblo  PBR2  Range. 
These  datasets  were  provided  to  SAIC  on  12  October  2007.  Figure  1  shows  the  (blue)  perimeter 
outline  of  the  survey  area  overlaying  a  topographic  map  of  the  Range.  Also  shown  in  this  image 
are  the  blanket  coverage  areas  from  the  vehicular  MTADS  as  green  rectangles  and  the  widely- 
spaced  survey  transect  tracks  of  the  vehicular  array.  Magnetic  anomalies  along  the  transects  are 
noted  with  blue,  green  or  yellow  symbols.  The  vehicular  blanket  survey  coverage  areas  were 
used  in  this  project,  the  transect  vehicular  survey  data  were  not. 

Figure  2  shows  a  closer  image  of  the  three  upper  MTADS  blanket  survey  coverage  areas  from 
Figure  1.  These  magnetic  dipole  anomaly  images  are  superimposed  on  a  high  resolution  aerial 
photograph  of  the  area.  The  purple  circle  overlays  the  outer  ring  of  an  aiming  bull’s  eye.  Also 
visible  are  partial  remnants  of  some  of  the  inner  circles  in  the  bull’s  eye.  As  is  typical,  these  are 
spaced  at  100  foot  intervals.  The  left  magnetic  survey  image  in  Figure  2  shows  a  relatively 
higher  density  of  anomalies  nearer  the  center  of  the  bull’s  eye.  These  anomalies  decrease  in 
density  with  distance  from  the  center  of  the  aiming  circle. 

Figure  3  illustrates  the  extent  of  the  airborne  magnetometer  survey  that  was  conducted.  Because 
of  the  rough  and  uneven  terrain  in  the  northwest  quadrant  of  the  planned  survey  area,  it  was  not 
mapped.  The  images  in  Figure  3  are  shown  as  Analytic  Signal  presentations.  The  expanded 
image  from  the  southwest  quadrant  of  the  survey  shows  an  additional  bull’s  eye  aiming  point 
denoted  by  relatively  high  concentration  of  magnetic  anomalies.  It  is  also  apparent  that  a  fence 
line  and/or  a  road  crosses  east-to-west  just  north  of  the  bull’s  eye.  Data  were  not  provided  to 
SAIC  as  images.  We  requested  that  only  minimally  processed  sensor  data  be  provided  to  us. 
This  is  usually  referred  to  as  pre-processed  data,  which  consists  of  removing  data  from  sensor 
platform  turns  and  areas  without  RTK  GPS  coverage  or  areas  where  the  helicopter  altitude  was 
too  high  to  provide  useful  information.  We  describe  below  the  form  of  data  that  was  actually 
provided. 
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Figure  1.  Image  of  the  survey  area  of 
PBR2.  The  perimeter  is  shown  in  blue. 
The  10  blanket  MTADS  vehicular  survey 
areas  appear  as  green  rectangles.  The 
transect  surveys  are  shown  as  green 
(course-over-ground  tracks)  with 
magnetic  anomalies  noted  by  various 
symbols. 
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Figure  2.  This  image  shows  more  detail  of  the  three  vehicular  survey 
areas  shown  at  the  top  of  Figure  1.  Other  details  are  discussed  in  the 
text. 
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Figure  3.  The  airborne  survey  area  at  PBR-2  is  shown  in  yellow  in  the  right  image  overlaying  a 
topographic  map  of  the  area.  A  16  ha  area  in  the  southwest  quadrant  is  shown  on  an  expanded  scale  on 
the  left.  This  is  an  Analytic  Signal  image  that  shows  a  concentration  of  targets  typical  of  a  target  bull’s 
eye.  Note  the  fence  line  slightly  north  of  the  bull’s  eye. 


2.0  The  PBR-2  Datasets 

Both  the  vehicular  and  airborne  data  provided  to  us  had  been  preprocessed  using  Geosoft  montaj 
utilities.  Both  the  vehicular  and  airborne  data  had  non-linear  spline  filtering  applied.  This  is  a 
down-the -track  smoothing  filter  that  removes  baseline  drifts  in  the  sensor  readings  (and  temporal 
changes  in  the  Earth’s  field  magnitude).  Additionally,  the  data  had  been  leveled.  This  process 
removes  offsets  in  the  zero  value  readings  of  the  sensors  in  the  array.  Data  taken  in  turns  had 
also  been  removed. 

The  vehicular  survey  data  had  additionally  been  processed  by  interpolating  the  data  onto  a  l/8th 
meter  grid. 

The  airborne  data  were  not  interpolated  when  we  received  them.  Airborne  data  were  provided  as 
a  mapped  data  file.  Each  individual  airborne  sensor  reading  was  accompanied  by  five  other 
values  including  the  GPS  clock  time  of  the  reading,  the  X  and  Y  coordinates  (UTM  in  meters), 
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Height  Above  Ellipsoid  (HAE  in  meters),  and  the  sensor  array  altitude  above  the  ground  (in 
meters). 

SAIC  reprocessed  both  the  vehicular  and  airborne  datasets  using  Geosoft  montaj  utilities.  The 
interpolated  vehicular  data  were  recast  in  an  Analytic  Signal  format  and  mapped  onto  a  %  meter 
grid.  Digital  images  were  provided  to  VLS  at  several  different  (described  below)  presentation 
scales. 

The  Airborne  data  were  recast  in  an  Analytic  Signal  format  and  mapped  onto  a  V2  meter  grid. 
Digital  images  were  provided  to  VLS  at  several  different  (described  below)  presentation  scales. 

3.0  Ground  Truth 
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We  inspected  the  PBR-2  datasets  for  data  quality  and  consistency,  variations  in  the  level  of 
geological  interferences,  and  the  density  of  anomalies.  We  examined  the  Archive  Search 
Information  available  to  us  to  determine  the  types  of  ordnance  deployed,  the  delivery  platforms, 
and  the  time  frames  in  which  the  Range  was  active.  We  have  numerous  survey  datasets 
available  to  us  with  similar  histories  to  PBR-2  and  for  which  there  were  extensive  dig  programs 
to  generate  ground  truth  following  the  surveys.  Both  vehicular  and  airborne  surveys  are 
available  for  the  Badlands 
Bombing  Range  and  the  Isleta 
Pueblo  S-l  Range.  We  have 
vehicular  survey  data  for  the  N9 
and  N10  Ranges  on  the  Laguna 
Pueblo  and  the  Buckley 
Bombing  Range.  Overall,  the 
best  match  for  the  PBR-2  Range 
was  determined  to  be  the 
Buckley  Range.  This  Airborne 
Impact  Range  contains 
predominately  M3 8s,  but  also 
contains  both  smaller  ordnance 
(down  to  M23s)  and  larger 
ordnance  (intermediate  sized 
cluster  bomb  units).  In  the 
northeast  comer  of  the  Buckley 
Survey,  a  1  hectare  area  (Sector 
A5)  was  analyzed  as  having 
nearly  400  targets.  All  targets  in 
this  sector  were  dug;  the  results 
provide  the  complete  ground 
truth  for  the  dig  list.  A  dipole 
image  presentation  of  this  area 
is  shown  in  Figure  4.  SAIC 
prepared  Analytic  Signal  digital 
images  of  Sector  A5  in  the  same 
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Figure  4.  Magnetic  anomaly  dipole  image  of  Sector  A5  on  the 
Buckley  Bombing  Range. 
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presentation  scales  that  were  prepared  for  the  vehicular  survey  areas  of  PBR-2.  These  were 
provided  to  SAIC  along  with  a  suggested  list  of  12  True  Positive  and  6  True  Negative  anomaly 
examples  for  training.  The  extract  from  the  Section  A5  dig  list  that  shows  the  suggested  training 
examples  is  shown  in  Table  1.  The  use  of  this  data  is  described  in  the  next  section  of  this 
report.  Only  M38s,  2.25  in  Rockets,  and  the  larger  Incendiary  Cluster  Bombs  were  chosen  as 
True  Positives.  At  this  site  the  ground  was  particularly  hard  and  many  of  the  M3 8s  were 
severely  deformed  (or  completely  fragmented)  on  impact.  One  M3  8  anomaly  was  also  chosen  as 
a  “True  Negative”  example  because  its  anomaly  signature  closely  resembled  a  dispersed  group 
of  clutter  items.  Many  other  ordnance  targets  from  the  site  were  also  excluded  from  the  training 
list  because  they  were  much  smaller  than  we  considered  as  typical  of  the  ordnance  dropped  on 
PBR2.  The  effect  that  this  had  on  the  overall  training  evaluation  using  the  Buckley  data  is 
discussed  in  the  next  section  of  this  report. 

Table  1.  Target  Report  from  Buckley  Bombing  Range,  Section  A5,  Training  Examples 


True  Positive  and  True  Negative  Examples  for  Training  for  Pueblo  Bombing  Range  #2 

True  Positive  Examples 
True  Negative  Examples 


Because  of  the  late  start  for  this  project  we  have  progressed  only  as  far  as  completing  Task  1  for 
the  single  site  PBR-2.  In  the  remainder  of  this  report  we  will  describe  the  approach  we  have 
taken,  the  results  that  were  generated  and  submitted  to  the  Program  Office  for  review,  and  our 
plans  for  completing  the  project. 
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4.0  Application  of  the  Feature  Analyst  Toolkit  to  the  PBR  Data 


For  the  completion  of  Task  1  as  stipulated  in  the  Statement  of  Work,  a  vehicular  target  model 
was  developed  to  apply  to  new  data  provided  by  the  Program  Office  and  processed  by  SAIC. 
Several  tests  were  conducted  using  the  most  effective  automated  target  models  previously 
developed  for  the  Badlands  Bombing  Range  (BBR)  project.  All  models  were  then  applied  to  the 
Buckley  Bombing  Target  data  in  a  batch  processing  mode.  The  Target  Picker  was  exercised  to 
generate  an  initial  list  of  potential  UXO  candidates  and  the  Target  Ranker  was  then  used  to 
classify  and  rank  the  potential  UXO  candidates.  Using  these  ranked  candidates,  combined  with 
the  ground  truth  from  the  Buckley  Impact  Range  provided  by  SAIC  (Table  1),  an  accuracy 
assessment  of  each  model  was  performed  to  determine  which  was  the  most  effective.  The 
preferred  model  was  then  set  aside  as  a  template  or  Target  Model  File  to  be  further  refined  and 
applied  later  to  the  Pueblo  PBR2  Range. 

Before  proceeding  to  the  PBR  data,  it  was  determined  that  some  existing  tools  in  the  Feature 
Analyst  UXO  Toolkit  could  be  improved.  In  addition  several  new  tools  were  identified  and 
implemented  to  improve  the  results  and  streamline  the  overall  process.  Once  the  new  software 
improvements  were  in  place,  the  automated  target  model  was  re-applied  to  the  Buckley  Bombing 
Target;  this  time  incorporating  the  image- wide  segmentation  tool,  which  was  extensively 
described  earlier.  Upon  analyzing  the  results,  it  was  determined  that  the  use  of  image-wide 
segmentation  process  dramatically  improved  the  accuracy  of  the  results.  This  improvement  in 
accuracy  was  based  on  (1)  visible  differences  in  anomaly  signatures  before  and  after 
segmentation  and  (2)  the  higher  rankings  for  problematic  UXO  candidates  as  compared  to  the 
tests  run  before  segmentation.  Additionally,  a  measurable  improvement  of  the  true  positive  ratio 
in  the  ROC  curve  was  noted.  These  developments  are  described  below  in  the  Section  titled 
Buckley  Testing  Results. 

5.0  Feature  Analyst  Workflow  Used  for  Task  1 

Below  is  a  list  of  the  steps  that  we  took  to  accomplish  the  goals  for  the  Task  1  deliverable. 

1.  Collect  and  apply  previous  Target  Models  from  the  Badlands  Bombing  Range  to  the 
Buckley  Bombing  Target  data  using  the  provided  ground  truth. 

2.  Analyze  the  results  and  determine  which  models  were  most  effective  in  accurately 
detecting  UXO  candidates  based  on  provided  ground  truth. 

3.  Develop  and  refine  new  and  existing  software  tools  (described  below)  in  an  effort  to 
improve  previous  results. 

4.  Re-apply  the  models  and  analyze  the  results  to  determine  if  software  refinements  had  any 
impact  on  accuracy. 

5.  Evaluate  the  available  Feature  Analyst  parameter  settings  to  refine  and  improve  the 
results. 

6.  Run  several  iterations  of  the  “Classify  Shapes  by  Probability”  tool,  each  time  adding 
additional  true  positive  and  true  negative  examples.  The  first  iteration  included  a  small, 
subset  of  the  provided  True  Positive  and  True  Negative  examples.  In  subsequent 
iterations,  the  Target  Ranker  was  iteratively  retrained  by  adding  equal  ratios  of  True 
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Positive  and  True  Negative  examples  from  the  remaining  list  of  candidates  from  Table  1 
(based  on  the  best-ranked  candidates  from  the  previous  iteration)  until  all  ground  truth 
examples  were  included  for  training  the  Target  Ranker.  Each  resulting  model  was 
analyzed  to  determine  whether  it  was  the  most  effective  in  finding  and  ranking  potential 
UXO  candidates.  The  results  are  described  below. 

7.  Using  the  most  effective  model,  generate  the  final  ranked  UXO  candidate  lists  for  PBR 
vehicular  areas. 

8.  Clip  areas  from  aerial  survey  dataset  where  the  aerial  image  overlaps  the  vehicular  image, 
then  generate  ranked  UXO  candidate  lists  corresponding  to  the  aerial  clips.  (NOTE:  Area 
2A  aerial  clips  were  split  into  two  sections  to  better  represent  the  sections  provided  in  the 
vehicular  data.) 

6.0  Software  Improvements 

In  order  to  generate  the  best  possible  results,  additional  tools  were  implemented  and 
improvements  to  existing  tools  were  made. 

6.1  Improvements  to  Existing  Tools 

■  A  separate  toolbar  for  the  UXO  Toolkit  with  a  dropdown  menu  to  allow  for  easy 
access  to  its  associated  software  tools  was  created  and  implemented. 

■  We  developed  and  added  the  ability  to  use  the  “Segment  Shapes”  tool  as  an 
image-wide  process  that  users  can  apply  using  batch  classification.  (Previously, 
the  tool  could  only  be  used  on  individual  polygons,  which  required  direct  user 
interaction.) 

■  We  added  the  ability  to  combine  separate  models  in  Feature  Modeler.  This  is 
particularly  useful  for  adding  new  processes  (such  as  image  segmentation)  to 
existing  models. 

6.2  New  Tools 

■  Import  Points  Tool  -  This  tool  allows  users  to  import  delimited  text  files 
containing  point  locations  to  be  displayed  as  a  feature  layer. 

■  Label  Features  Tool  -  Given  a  ground  truth  layer  and  results  layer,  this  tool 
creates  a  new  attribute  column  called  “GmdTruth”  and  labels  individual  polygons 
as  true  positive,  false  positive,  false  negative,  or  unknown. 

■  ROC  Curve  Generator  -  Given  labeled  results  layers,  this  tool  generates 
associated  ROC  curves  for  analysis  and  evaluation. 

7.0  Buckley  Testing  Results 

All  target  models  from  the  Badlands  Bombing  Range  library  were  first  applied  to  each  of  the 
three  palate  ranges  for  the  Buckley  data.  As  described  in  the  Workflow  Description  section, 
several  iterations  of  Target  Ranker  were  performed  to  determine  whether  the  accuracy  of  the 
target  ranker  was  affected  by  adding  more  ground  truth  points  for  training.  The  last  model  (with 
all  ground  truth  points  included)  was  determined  to  have  the  highest  level  of  accuracy  in  ranking 
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potential  UXO  candidates.  The  criteria  used  to  determine  the  level  of  performance  of  the  Target 
Ranker  were  visual  analysis,  the  individual  target  rankings,  and  true  positive  rates  based  on  ROC 
Curve  data.  A  comparison  of  the  ROC  curves  produced  by  the  Target  Ranker  using  increasing 
numbers  of  training  examples  is  shown  in  Figure  5. 
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Figure  5.  ROC  curves  showing  the  relative  performance  of  the  Target  Ranker.  With  each  iteration,  the 
Target  Ranker  was  re-trained  using  additional  ground  truth  examples.  With  more  ground  truth 
examples  the  overall  accuracy  visibly  increases.  The  areas  under  the  ROC  curves  for  the  first  through 
fifth  iterations  were  105.9, 111.9, 112, 115.1  and  116.9  respectively. 


Once  the  preferred  target  model  developed  from  BBR  evaluations  was  selected 
(BT2_Section5_300_Revised_Target_Model_Results_Fifth.afe),  several  additional  tests  were 
conducted  on  each  Buckley  palette  range  using  a  new  variation  of  the  Shape  Segmentation  Tool 
to  further  enhance  the  results.  Developing  and  applying  the  Segmentation  Tool  did  indeed  make 
significant  improvements  in  the  performance  of  the  Target  Ranker  using  the  modified  ROIs.  In 
several  instances,  application  of  the  segmentation  tool  managed  to  separate  True  Positive  targets 
examples  that  were  merged  with  possible  non-UXO.  In  Figure  6,  we  show  an  example  of  this 
process.  In  this  graphic,  the  bottom  layer  of  the  image  is  a  clip  from  the  false-color  digitized 
image  (of  the  Analytic  Signal  ±300  nT/M  palate  provided  by  SAIC  from  the  Buckley  A5 
dataset).  On  the  next  layer  up,  the  ROIs  specified  by  the  Target  Picker  are  shown  in  Light  Blue; 
these  were  chosen  without  use  of  the  Segmentation  Tool.  The  very  small  black  diamond  in  the 
upper  part  of  the  large  center  blob  is  the  coordinate  position  of  one  of  the  provided  True  Positive 
Training  Examples. 
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Following  the  implementation  of  the 
Segmentation  Tool,  the  ROIs  generated  by  the 
Target  Picker  are  shown  as  the  light  pink 
features.  These  form  the  third  layer  in  the 
graphic  image.  Application  of  the 
Segmentation  Tool  has  effectively  isolated  the 
ROI  associated  with  the  True  Positive 
Anomaly  Signature.  As  shown  in  the  image, 
following  application  of  the  Segmentation 
Tool,  this  True  Positive  example  is  an 
excellent  choice  for  the  learner  to  apply  in  the 
Target  Ranking. 


Figure  6.  This  image  shows  an  example  of  the 
effects  of  application  of  the  segmentation  tool.  The 
small  diamond  in  the  upper  center  marks  the 
position  of  one  of  the  True  Positive  training 
examples.  The  area  of  the  entire  image  clip  is  10 
square  meters.  See  the  text  for  details. 


A  legitimate  question  remains  about  the 
disposition  of  the  anomaly  shape  immediately 
below  the  “True  Positive”  example  in  the 
lower  portion  of  Figure  6.  This  feature  may  be 
a  non-ordnance  anomaly  based  upon  its 
oblong  shape  and  dimensions,  or  it  might  be 
two  features,  which  have  not  been  separated  at 
the  ±300  nT/m  presentation  scale  used  for  this 
analysis.  This  is  typical  of  the  type  of  feature 
that  ultimately  will  require  a  second  review 
following  the  application  of  Feature  Analyst. 
Figure  7,  which  is  a  dipole  presentation  of  the 
same  clip  at  a  palate  scale  of  ±70  nT,  shows 
that  the  lower  feature  is  indeed  two  separate 
anomalies.  Note  also  that  at  the  presentation 
scale  required  to  separate  these  images  in 
Figure  7,  that  several  of  the  smaller  features 
shown  in  Figure  6  are  lost.  Because  we  use  a 
single  digital  image  palate  for  the  Feature 
Analyst  classification,  these  smaller 
(potentially  UXO)  features  must  be  preserved 
in  the  image. 

The  final  results  with  the  Buckley  Section  A5 
Training  data  and  using  the  automated 
Segmentation  Tool  are  provided  below: 


Results  for 

BT2_Section5_300_Revised_Target_Model_Results_Fifth.afe 

Total  ROI’s:  399 
Threshold:  0.30 
ROI’s  above  the  threshold:  273 


Figure  7.  This  is  a  dipole  image  clip  of  the  same 
area  shown  in  Figure  6.  In  this  image  the  palate 
scale  is  plus  or  minus  70  nT. 
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The  Ground  Truth  for  the  Buckley  Bombing  Range  Section  A5  is  shown  in  an  Excel 
Spreadsheet,  which  is  included  in  electronic  format  in  the  Appendix.  In  this  spreadsheet  we  have 
rationalized  the  Feature  Analyst  analysis  with  the  original  MTADS  analysis  and  the  Ground 
Truth  recorded  by  the  dig  teams  when  the  targets  were  excavated.  There  were  386  targets  dug  in 
this  survey  section.  Based  upon  the  diggers  declarations  there  were  321  targets  that  were  intact 
UXO,  UXO  components,  and  UXO  “Scrap.”  Numerous  of  the  diggers  declarations  were 
ambiguous,  and  many  UXO  declarations  were  for  ordnance  items  that  were  smaller  (M23s, 
fuzes,  etc)  than  those  that  were  included  in  the  True  Positive  training  list  for  use  with  Feature 
Analyst.  The  Feature  Analyst  Learner, 
using  the  parameters  specified  above 
correctly  identified  241  UXO  items,  the 
Target  Picker  missed  65  UXO  items, 
and  the  Classifier  misclassified  15  UXO 
as  not-UXO.  These  results  are  shown  in 
the  ROC  curves  in  Figure  8. 


The  15  targets  that  were  misclassified 
were  not  of  concern.  They  were 
primarily  targets  that  were  smaller  than 
the  ordnance  included  as  True  Positive 
examples  for  training  the  learner.  Also 
included  were  a  few  M3 8s  which  were 
so  distorted  or  fragmented  that  they 
appeared  to  be  clutter. 

Of  the  targets  missed  by  the  Target 
Picker,  some  were  smaller  than  those 
that  the  learner  was  trained  to  recognize 
as  ordnance.  Many  others  however, 
were  lost  in  large  anomaly  shapes  that 
consisted  of  multiple  overlapping 
anomaly  signatures.  Some  of  these 
clustered  ROIs  remained  in  spite  of  the 
application  of  the  Segmentation  Tool. 
These  “lost”  ROIs  are  worrisome 
because  they  contribute  both  to  missed 
targets  and  to  misclassified  ROIs.  This 
issue  is  discussed  further  below. 
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Figure  8.  ROC  Curves  demonstrating  the  performance 
of  the  Feature  Analyst  Learner  on  the  Buckley  Section  A5 
survey  area. 
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8.0  Palate  Analysis  Issues 

Before  beginning  discussion  of  the  PBR  analysis,  we 
would  like  to  point  out  and  discuss  three  types  of 
troublesome  anomaly  signatures  that  require  special 
attention  in  the  analysis  and  in  reviewing  the  results.  The 
first  of  these,  are  small  polygons  with  weak  magnetic 
signatures  that  were  listed  as  True  Positive  training 
examples  in  the  Buckley  training  examples.  One  of  these 
is  shown  in  Figure  9.  Working  with  either  the  ±250  or 
±300  nT/m  palates,  the  Target  Picker  was  unable  to 
capture  the  anomaly  This  True  Positive  training  example 
was  a  2.25  in  rocket.  It  was  relatively  deeply  buried  and 
had  a  magnetic  moment  less  than  half  of  the  next  larger 
training  example.  It  was  intended  to  represent  the  smallest 
likely  potential  ordnance  target.  Based  upon  the  smaller 
True  Positive  examples  to  be  provided  by  the  Program 
Office  for  the  PBR2  Task  2  analysis,  we  may  have  to 
either  reconsider  the  palate  range  for  the  digital  analysis 
image,  or  to  run  the  whole  process  in  an  additional  pass 
with  a  more  sensitive  image  scale  to  separately  capture  the 
smallest  true  ordnance. 

A  second  type  of  anomaly  that  creates  analysis  problems 
involves  True  Negative  examples  that  greatly  resemble 
True  Positive  examples  (see  Figure  10).  As  a  result,  these 
true  negative  examples  are  extremely  difficult  for  the 
Target  Ranker  to  distinguish  from  true  positives  and 
therefore,  they  are  typically  ranked  very  high.  Because  of 
the  way  many  M3 8s  either  distort  or  break  up  on  impact, 
they  create  distorted  signatures  that  may  resemble 
geological  returns,  bundles  of  fence  wire,  etc.  This  creates 
an  analysis  problem  both  for  this  site  and  for  many  other 
sites  that  contain  similar  challenges. 

Thirdly,  some  polygons  containing  multiple  overlapping  anomaly  signatures  are  extremely 
difficult  or  impossible  to  segment.  Occasionally,  these  polygons  such  as  the  one  shown  in 
Figure  11  contain  both  true  positive  and  true  negative  ground  truth  examples.  This  ±300  nT/m 
AS  image  is  at  too  fine  a  scale  to  segregate  these  overlapping  anomalies.  Figure  12  shows  a 
dipole  image  at  a  120  nT  scale  that  isolates  the  overlapping  features.  In  order  to  isolate  these 
features  as  an  Analytic  Signal  digital  graphic  would  require  a  significantly  courser  palate 
(perhaps  ±450  nT/m). 

The  examples  shown  in  Figures  9  and  11  show  that  the  +300  nT/m  palate  chosen  for  the 
vehicular  analysis  has  problems  with  targets  both  at  the  largest  and  smallest  ends  of  the  size 
spectrum.  These  problems  are  further  exacerbated  when  analyzing  airborne  data  because  the 
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Figure  9.  This  image  shows  a  2.5 
meter  square  clip  from  the  Analytic 
Signal  Buckley  Image  used  for 
training  the  learner.  The  circle 
marks  the  position  of  Training 
Target  No.  33  in  Table  1. 


Figure  10.  This  9  X  9  m  image  clip 
from  the  Buckley  training  site 
contains  two  training  examples  (one 
positive  and  one  negative)  that 
greatly  resemble  each  other. 
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Figure  11.  This  6  X  6  m  image  clip  from  the 
Buckley  training  site  contains  4  positive  and 
negative  training  examples.  Three  of  them 
cannot  be  isolated  using  the  Segmentation  Tool  at 
this  palate  scale. 


Figure  12.  This  dipole  image  clip  displayed  at  a 
palate  scale  of  plus  or  minus  120  nT  shows 
approximately  the  same  area  as  Figure  10.  In 
this  image  the  anomalies  area  effectively 
separated. 


anomaly  footprints  are  typically  much  larger  than  the  vehicular-measured  anomalies  where  the 
sensors  are  placed  much  closer  to  the  ground.  Depending  upon  the  size  range  of  the  True 
Positive  examples  provided  by  the  Program  Office  for  PBR2,  we  will  likely  come  up  with  an 
analysis  strategy  that  involves  additional  analysis  passes  with  the  data. 


9.0  Analysis  of  the  Pueblo  PBR2  Datasets 

During  the  first  round  of  Buckley  dataset  testing  where  the  shape  segmentation  tool  was  not 
used,  it  was  determined  that  palate  range  -250  to  250  yielded  the  best  results  with  the  preferred 
target  model.  Still,  some  troublesome  true  positive  UXO  remained  ranked  lower  than  all  false 
positive  targets  (including  the  candidate  Figure  6).  When  the  Shape  Segmentation  Tool  was 
applied,  the  best  results  were  produced  using  the  palate  range  ±300.  This  likely  occurred  because 
the  targets  were  better-define;  furthermore,  these  images  contained  less  of  the  small  clutter  areas 
that  could  be  potentially  misclassified.  These  characteristics  were  also  instrumental  in  the 
success  of  the  shape  Segmentation  Tool  and  its  ability  to  correctly  isolate  positive  candidates 
from  adjacent  merged  anomaly  signatures.  Except  for  the  implementation  of  the  shape 
segmentation  process,  no  adjustments  were  made  to  the  learning  settings  of  the  original  model 
that  we  had  developed  and  applied  to  datasets  from  the  BBR.  These  settings  are  provided  below: 

Learning  Settings: 


•  Bands:  (3)  BT2_Section5  palette  range  -300  to  300.tif 

•  Input  Representation:  Square  5x5 

•  Approach:  1 

•  Aggregation:  5 

•  Smoothing:  Post  processing  Bezier  Smoothing  step 
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•  Histogram  Stretch:  No 

•  Rotation:  Yes 

•  Clutter  by  shape:  All  options  selected 

10.0  PBR  Analysis  Results 

Because  the  best  results  from  the  Buckley  Range  data  came  from  the  ±300  nT/m  palate  range, 
the  final  target  model  was  applied  to  the  PBR2  vehicular  data  with  the  same  range.  It  was  also 
determined  that  the  most  consistent  results  for  the  aerial  data  were  generated  using  the  imagery 
with  the  palate  range  ±16  nT/m.  With  the  exception  of  ground  truth  data,  the  same  criteria 
previously  used  for  judging  target  accuracy  on  Buckley  Range  were  used  for  analyzing  the  PBR 
datasets.  Additionally,  given  the  absence  of  ground-truth  data  specifically  from  this  site,  target 
UXO  candidates  were  visually  inspected  to  be  sure  their  associated  ranks  were  consistent  across 
all  parts  of  the  survey  area. 

Finally,  after  examining  the  results  of  the  analysis,  a  cutoff  was  established  for  the  UXO 
probability  ranking  below  which  we  feel  that  ROIs  can  be  safely  declared  as  non-UXO.  Some 
highly-ranked  candidates  in  the  vehicular  data  are  less  distinct  in  the  aerial  imagery,  and  some  of 
the  smaller  polygons  in  the  vehicular  examples  were  simply  not  present  in  the  airborne  data. 
Even  taking  into  account  these  differences  in  available  polygon  detail  and  the  resulting  variation 
in  respective  polygon  rankings,  we  have  established  the  same  thresholds  for  the  vehicular  and 
airborne  surveys.  The  threshold  probability  value  was  set  to  the  rather  low  value  of  0.20.  The 
over-riding  factor  here  was  that  we  were  applying  ground  truth  for  a  site  that  had  a  different 
history  than  the  PBR2  Range.  Hopefully,  we  can  be  more  discriminating  when  we  rerun  the 
training  process  using  real  True  Positive  and  True  Negative  examples  recovered  from  the  Range. 

A  summary  of  our  analyses  of  the  ten  separate  survey  areas  is  provided  in  Table  2.  In  the 
airborne  dataset  we  split  the  analysis  of  Area  A2  into  two  sections  to  conform  to  the  way  that  the 
vehicular  data  were  collected.  The  21  Excel  spreadsheets  with  entire  listings  of  ROIs  and  their 
rankings  are  provided  in  electronic  format  in  the  Appendix.  In  all  10  areas  of  the  vehicular 
survey  area  the  Target  Picker  specified  4291  ROIs.  With  the  0.2  probability  threshold,  we 
declared  3529  of  these  as  potentially  UXO. 

The  corresponding  areas  of  the  airborne  survey  dataset  were  analyzed  with  the  Target  Picker 
declaring  991  ROIs.  A  total  of  914  of  these  were  declared  as  potential  UXO  based  upon  the 
probabilities  calculated  by  the  Target  Ranker. 

The  significantly  smaller  number  of  ROIs  identified  from  the  airborne  dataset  were  the  result  of 
the  “disappearance”  of  many  of  the  smaller  anomalies  in  the  airborne  data.  In  addition,  a 
significant  number  of  the  anomaly  signatures  merged  with  each  other  in  the  airborne  data.  The 
resulting  anomaly  shapes  and  magnetic  signal  intensity  distributions  led  the  Target  Picker  to 
disregard  them  in  the  initial  analysis  pass.  The  fact  that  the  learner  was  also  trained  only  on 
vehicular  data  may  have  played  a  part  in  the  Target  Picker  having  rejected  a  large  number  of 
anomalies.  We  had  discussed,  but  did  not  implement  the  plan  to  take  a  few  of  the  highest  ranked 
(and  will-isolated)  targets  from  the  vehicular  survey  analysis  and  using  their  corresponding 
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signatures  (as  true  positives)  to  train  the  learner  for  the  airborne  survey.  These  issues  will  be 
revisited  before  we  begin  the  Task  2  analysis. 


Table  2.  Summary  of  the  vehicular  and  airborne  analysis  results  for  the 
ten  survey  areas  of  PBR2 


Vehicular  Survey,  Analytic  Signal  Image,  Pallate  ±  300  nT/m 

Survey  Area 

Total  Identified 
ROIs 

Declaration 

Threshold 

Probability 

Declared 

UXO 

Area  1A 

107 

0.20 

64 

Area  2A 

143 

0.20 

97 

Area  3A 

672 

0.20 

564 

Area  2A 

211 

0.20 

101 

Area  2B 

111 

0.20 

42 

Area  3A 

1449 

0.20 

1320 

Area  3B 

409 

0.20 

337 

Area  3C 

167 

0.20 

110 

Area  BT4  Center 

936 

0.20 

831 

The  Simmons  Area 

86 

0.20 

63 

Total  Vehicular  Survey 
Declarations 

4291 

3529 

Airborne  Survey,  Analytic  Signal  Image,  Pallate  ±16  nT/m 

Survey  Area 

Total  Identified 
ROIs 

Declaration 

Threshold 

Probability 

Declared 

UXO 

Area  1A 

25 

0.20 

22 

Area  2A,  Section  1 

38 

0.20 

37 

Area  2A  Section  2 

25 

0.20 

11 

Area  3A 

72 

0.20 

66 

Area  2A 

22 

0.20 

13 

Area  2B 

42 

0.20 

24 

Area  3A 

531 

0.20 

516 

Area  3B 

69 

0.20 

65 

Area  3C 

25 

0.20 

24 

Area  BT4  Center 

109 

0.20 

106 

The  Simmons  Area 

33 

0.20 

30 

Total  Airborne 

Survey  Declarations 

991 

914 
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11.0  Summary 


Based  on  the  results  of  Task  1,  it  has  been  determined  that  the  original  BBR  target  model 
produced  acceptable  results  on  similar  but  separate  datasets  and  palate  ranges.  In  particular,  the 
Task  1  results  confirm  that  a  broader  palate  range  (coarser  palate  scale)  tends  to  yield  the  better 
results. 

Before  beginning  Task  2,  we  will  consider  the  effect  of  extending  the  digital  images  to  even 
coarser  palate  scales  for  both  the  vehicular  and  airborne  datasets.  The  goal  of  this  process  would 
be  to  “unmerge”  overlapping  anomaly  signatures,  or  sufficiently  uncoupling  them  that  the 
Segmentation  Tools  could  isolate  them.  A  natural  consequence  of  this  process  is  that  we  will 
have  to  implement  an  additional  process  using  a  much  finer  resolution  palate  scale  for  an 
additional  analysis  pass  with  the  Learner. 

Probably  the  overall  analysis  would  be  done  by  using  the  coarser  palate  to  carry  out  the  primary 
analysis.  Following  this,  we  would  visually  inspect  several  finer  scale  images  and  flag  new 
targets  with  intensities  that  raised  them  above  the  noise  floor  sufficiently  that  the  Target  Picker 
could  capture  them.  The  goal  of  this  step  would  be  to  clearly  capture  all  the  known  True 
Positives.  These  finer-scale  targets  would  have  to  be  flagged  and  followed  through  the  second 
pass  with  both  the  Target  Picker  and  Target  Ranker.  The  newly  Ranked  (Flagged)  target  list 
would  then  have  to  be  rationalized  by  hand  with  the  initial  list  to  produce  a  single  result. 

We  also  established  during  this  year’s  work  that  with  the  addition  of  an  image-wide  shape 
segmentation  step,  our  previous  results  could  be  significantly  improved  while  at  the  same  time 
reducing  the  amount  of  required  user  interaction  during  the  preliminary  steps  of  the  analysis. 
Additionally,  the  other  previously-described  software  enhancements  served  to  further  streamline 
the  process. 

For  the  next  task,  which  will  include  actual  ground  truth  for  the  PBR  datasets,  it  will  be  possible 
to  test  and  further  improve  our  new  and  existing  tools  while  at  the  same  time  experiment  with 
parameter  settings  in  an  effort  to  reduce  more  clutter  and  better  rank  UXO  candidates  with  an 
even  higher  level  of  confidence. 
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